A concept in selective breeding and zoology, outbreeding depression refers to cases when offspring from crosses between individuals from different populations have lower fitness than progeny from crosses between individuals from the same population. (In selective breeding, the concept is opposed to inbreeding depression). This phenomenon can occur in two ways.
First, selection in one population might produce a large body size, whereas in another population small body size might be more advantageous. Gene flow between these populations may lead to individuals with intermediate body sizes, which may not be adaptive in either population. It might be that, in a certain environment, having either a large or a small body is advantageous, whereas an intermediate-sized body is comparatively disadvantageous.
A second way outbreeding depression can occur is by the breakdown of biochemical or physiological compatibilities between genes in the different breeding populations. Within local, isolated breeding populations, alleles are selected for their positive, overall effects on the local genetic background. Due to nonadditive gene action, the same genes may have rather different average effects in different genetic backgrounds--hence, the potential evolution of locally coadapted gene complexes. However, on the other hand, according to the overdominance hypothesis in genetics, it is believed that certain combinations of alleles (which can only be obtained by outbreeding) are especially advantageous when paired in a heterozygous individual, which is one explanation for the existence of hybrid vigor.
A third, but neutral, effect of outbreeding is the loss of allopatric speciation of a particular group, that lends its distinctness and contributes to the diversity of said types in either group by the exclusive retention of select traits.
Considering the second manner of outbreeding depressing cited above, individuals from Population A will tend to have alleles selected for the quality of combining well with allele combinations common in Population A. However, alleles found in Population A will not have been selected for the quality of crossing well with alleles common in Population B. Therefore outbreeding can undermine vitality by reducing positive epistasis and/or increasing negative epistasis. The sterility and other fitness-reducing effects often seen in interspecific hybrids (such as mules) can be considered an extreme case of this type of outbreeding depression, involving not only different alleles of the same gene (as in distinct populations of a single species) but even different orthologous genes.
However, it is critical to understand that reduced inbreeding depression in first generation hybrids can, in some circumstances, be strong enough to more than make up for outbreeding depression. Because of this and because of the uniformity and predictable outcome of a first generation hybrid (F1 hybrid) farmers keep purebred strains for the purpose of outcrossing. Crossing the hybrids will give unpredictable outcomes and outbreeding depression will remain or worsen so that is not common practice.
As a general rule of thumb, hybrid vigor (another way of saying a reduction of inbreeding depression) is strongest in first generation hybrids and gets weaker over time. In contrast, outbreeding depression can be relatively weak in the first generation. But outside the context of ruthless selective pressure, outbreeding depression will increase in power through the further generations as co-adapted gene complexes are broken apart without the forging of new co-adapted gene complexes to take their place.
It is important to keep in mind that these two mechanisms of outbreeding depression can be operating at the same time. However, determining which mechanism is more important in a particular population is very difficult.